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Abstract 

One  year  after  its  initial  meeting,  the  Glycemia  Modeling  Working  Group  reconvened  during  the  2009  Diabetes 
Technology  Meeting  in  San  Francisco,  CA.  The  discussion,  involving  39  scientists,  again  focused  on  the  need 
for  individual  investigators  to  have  access  to  the  clinical  data  required  to  develop  and  refine  models  of  glucose 
metabolism,  the  need  to  understand  the  differences  among  the  distinct  models  and  control  algorithms,  and 
the  significance  of  day-to-day  subject  variability.  The  key  conclusion  was  that  model-based  comparisons 
of  different  control  algorithms,  or  the  models  themselves,  are  limited  by  the  inability  to  access  individual 
model-patient  parameters.  It  was  widely  agreed  that  these  parameters,  as  opposed  to  the  average  parameters 
that  are  typically  reported,  are  necessary  to  perform  such  comparisons.  However,  the  prevailing  view  was 
that,  if  investigators  were  to  make  the  parameters  available,  it  would  limit  their  ability  (and  that  of  their 
institution)  to  benefit  from  the  invested  work  in  developing  their  models.  A  general  agreement  was  reached 
regarding  the  importance  of  each  model  having  an  insulin  pharmacokinetic/pharmacodynamic  profile  that  is 
not  different  from  profiles  reported  in  the  literature  (88%  of  the  respondents  agreed  that  the  model  should 
have  similar  curves  or  be  analyzed  separately)  and  the  importance  of  capturing  intraday  variance  in  insulin 
sensitivity  (91%  of  the  respondents  indicated  that  this  could  result  in  changes  in  fasting  glucose  of  >15%, 
with  52%  of  the  respondents  believing  that  the  variability  could  effect  changes  of  >30%).  Seventy-six  percent 
of  the  participants  indicated  that  high-fat  meals  were  thought  to  effect  changes  in  other  model  parameters 
in  addition  to  gastric  emptying.  There  was  also  widespread  consensus  as  to  how  a  closed-loop  controller 
should  respond  to  day-to-day  changes  in  model  parameters  (with  76%  of  the  participants  indicating  that  fasting 
glucose  should  be  within  15%  of  target,  with  30%  of  the  participants  believing  that  it  should  be  at  target). 
The  group  was  evenly  divided  as  to  whether  the  glucose  sensor  per  se  continues  to  be  the  major  obstacle  in 
achieving  closed-loop  control.  Finally,  virtually  all  participants  agreed  that  a  future  two-day  workshop  should 
be  organized  to  compare,  contrast,  and  understand  the  differences  among  the  different  models  and  control 
algorithms. 
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Introduction 

T 

JLhe  Glycemia  Modeling  Working  Group  (GMWG)  was 
originally  formed  in  2008  with  the  intent  to  document 
current  practices  in  metabolic  modeling  for  advancing 
closed-loop  glucose  control.  The  intent  was  to  formulate 
a  framework  for  the  exchange  of  information  and 
collaboration  among  research  centers,  to  identify  limitations 
of  glycemia  models,  and  to  identify  essential  tasks  that 
need  to  be  accomplished  in  order  to  achieve  a  fully 
automated  closed-loop  glucose  control  system.  The  working 
group,  which  was  then  composed  of  25  scientists 
representing  many  of  the  world  leaders  in  glycemia 
modeling,  was  reconvened  in  2009  with  39  scientists  in 
attendance,  including  14  of  the  original  25  participants. 
This  report  serves  to  update  the  progress  achieved  on 
the  2008  recommendations1  and  summarizes  the  topics 
discussed  in  the  2009  meeting.  As  with  the  2008  report, 
we  sought  to  document  the  collective  efforts  and  view 
points  of  the  world  leaders  in  glycemia  modeling,  to 
increase  awareness  of  the  existing  modeling  challenges 
and  opportunities,  and  to  provide  information  to  guide 
funding  priorities  and  future  solicitations. 

Progress  Achieved  on  the  2008 
Recommendations 

Briefly,  the  main  conclusions  and  recommendations 
from  the  2008  GMWG  meeting  were  as  follows:1 

1.  Physiological  variance  represents  the  single  largest 
technical  challenge  to  creating  accurate  simulation  models. 

2.  A  Web  site  describing  the  different  models  and  data 
supporting  them  should  be  made  publicly  available, 
with  funding  agencies  and  journals  requiring 
investigators  to  provide  open  access  to  both  models 
and  data. 

3.  Existing  simulation  models  should  be  compared  and 
contrasted  using  the  same  evaluation  and  validation 
criteria  to  better  assess  the  state  of  the  art,  understand 
any  inherent  limitations  in  the  models,  and  identify 
gaps  in  data  and/or  model  capability. 

Although  it  was  generally  agreed  that  an  accurate 
description  of  physiological  variances  continue  to  be  a 
major  challenge  and  that  models  should  be  made  public 
(see  recommendations  1  and  2),  modest  progress  was 
reported.  Data  for  30  subjects  were  made  available  for 
a  model  developed  at  the  University  of  Virginia2  (UVA; 


simulator;  contact  UVA  Patent  Foundation  for  details). 
The  subjects — 10  adults,  10  adolescents,  and  10  children — 
were  selected  from  a  larger  virtual  population  of 
300  subjects  (100  in  each  group)  with  the  intent  to 
describe  a  wide  range  of  subjects,  including  both  "average" 
and  "extreme"  subjects  (with  the  extreme  subjects  used 
to  assess  the  robustness  of  different  control  algorithms). 
Intraday  variation  in  parameters  was  not  included,  as 
the  developers  do  not  believe  appropriate  data  exist 
for  their  identification  (personal  communication  from 
B.  Kovatchev).  Eighteen  subjects  were  also  identified 
for  a  model  developed  at  Cambridge  University,  with 
a  limited  subset  of  subject  data  available  from  the 
developer  under  a  research  license  agreement  (personal 
communication  from  R.  Hovorka).  The  Cambridge  model 
includes  intraday  variation  on  selected  model  parameters 
(5%  amplitude;  3-hour  period3).  Finally,  intraday  variances 
in  parameters  characterizing  a  model  developed  at 
Medtronic4  were  also  described  for  10  subjects  (intraday 
variation  in  model  parameters  identified  in  three  6-hour 
windows).  Nevertheless,  given  the  importance  placed  on 
understanding  how  intrasubject  variability  might  affect 
closed-loop  control  and  the  potential  for  models  to  aid 
in  the  development  of  effective  control  algorithms,  these 
are  small  steps.  Particularly  disappointing  is  that  no 
Web  site  was  created  describing  the  different  models 
(part  of  recommendation  2),  and  no  publication 
comparing  different  modeling  approaches  or  identifying 
any  limitations  or  gaps  has  appeared  (recommendation  3). 

The  prevailing  view  as  to  why  there  was  so  little  progress 
on  model  comparison  was  that  the  derivation  and 
validation  of  the  models,  as  well  as  the  identification 
of  the  underlying  parameters,  represent  a  substantial 
amount  of  work  by  the  individuals  who  developed 
them  and  that  making  such  information  freely  available 
to  other  investigators  or  companies  was  an  unrealistic 
objective.  The  models,  associated  simulation  software, 
and  any  related  control  algorithms  that  might  be  derived 
from  them  are  viewed  as  potential  sources  of  revenue 
through  intellectual  property  royalties  and  licensing 
agreements  or  as  providing  a  competitive  edge  when 
applying  for  extramural  funding.  At  issue  is  the  ability  to 
access  individual  parameter  sets  that  comprise  the  average 
parameter  values  typically  reported  in  publications. 
It  was  widely  agreed  that  simulations  performed  using 
the  average  parameter  values  do  not  yield  useful  results 
when  comparing  models  or  evaluating  control  algorithms, 
although  they  may  still  serve  in  a  more  limited  capacity. 
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Although  it  is  becoming  clear  that  individual  investigators 
are  unlikely  to  make  their  model  parameters  available, 
a  positive  outcome  of  the  meeting  was  that  virtually 
all  the  investigators  embraced  the  idea  of  a  subsequent 
Modeling  Comparison  Workshop  in  which  different 
models  would  be  compared  and  contrasted  using  common 
data  sets  and  testing  criteria.1  Also,  there  was  widespread 
agreement  on  what  we  would  like  to  accomplish 
with  the  individual  models — mainly  achieve  a  better 
understanding  of  the  strengths  and  weakness  of  each 
model  when  used  to  compare  and  evaluate  different 
control  algorithms.  There  was  also  substantial  agreement 
on  numerous  aspects  of  how  the  control  algorithm  should 
be  configured  and  introduced.  We  highlight  these 
discussions  next  and  return  to  the  issue  of  how  to  compare 
models  in  our  conclusion. 

Specific  Topics  Discussed  at  the  2009 
Glycemia  Modeling  Working  Group 
Meeting 

Similar  to  the  2008  GMWG  meeting,  a  series  of  questions 
was  used  to  guide  the  2009  meeting.  The  questions  and 
the  participants'  answers  are  provided  in  the  Appendix. 
Not  all  responses  add  up  to  39,  as  some  participants 
left  some  questions  blank  and  the  answers  from  the 
authors  were  not  tabulated.  Some  questions  sought  to 
survey  opinions  as  to  the  state  of  glycemia  monitoring, 
modeling,  and  control  technologies  (by  definition  questions 
that  have  no  correct  or  incorrect  answer),  while  other 
questions  attempted  to  obtain  answers  that  are  not 
necessarily  known  but  that  could  be  easily  verified  using 
the  different  metabolic  models.  As  the  questions  were 
largely  predetermined  by  the  authors,  the  following 
discussion  undoubtedly  reflects  the  authors'  biases,  with 
the  hope  that  future  meetings  can  mitigate  this  issue  by 
having  participants  submit  their  own  questions  in  advance. 
With  this  caveat  in  mind,  the  following  summarizes  the 
discussions  and  our  analyses  of  the  tabulated  results. 

Generally,  the  working  group  remained  cautiously 
optimistic  that  glucose  sensor  technology  is  advancing, 
with  -19%  (5/27)  of  the  respondents  believing  that  closed- 
loop  control  can  be  achieved  with  today's  technology  and 
an  additional  41%  (11/27)  of  the  respondents  believing  that 
improvements  in  continuous  glucose  monitoring  (CGM) 
calibration  and  filter  routines  will  suffice.  Approximately 
60%  of  the  respondents  did  not  believe  that  interstitial 
fluid  (ISF)  glucose  is  substantially  delayed  compared 
with  plasma  glucose,  but  only  30%  of  the  respondents 
believed  that  the  maximum  rate  of  change  of  ISF  glucose 
is  well  characterized  (with  the  average  value  reported 


by  these  individuals  as  4  mg/dl/min).  However,  of  those 
that  believed  the  delay  is  <10-15  min,  50%  believed  that 
insulin  can  cause  changes  in  the  blood  (i.e.,  plasma)-to-ISF 
glucose  gradient.  Also,  50%  still  indicated  that  the  biggest 
impediment  to  the  development  of  an  artificial  pancreas 
is  the  ability  to  reliably  measure  glucose.  Of  the  50%  of 
the  respondents  who  did  not  believe  that  glucose  sensing 
is  the  problem,  the  predominant  sentiment  (60%  [9/13]  of 
the  respondents)  was  that  the  absence  of  a  high-fidelity 
predictive  model  is  the  main  impediment.  Surprisingly, 
only  4  of  26  respondents  believed  that  a  complete  under¬ 
standing  of  what  the  control  strategy  should  be  remains 
an  open  issue.  Unfortunately,  we  did  not  survey  these 
individuals  as  to  what  control  strategy  they  believed  should 
be  used;  however,  given  the  number  of  respondents  who 
indicated  that  the  largest  impediment  was  the  lack  of  a 
high-fidelity  predictive  model  suggests  that  most  would 
favor  some  form  of  model  predictive  control  (MPC). 

Arguably,  MPC  should  always  be  the  preferred  approach 
in  instances  where  a  model  can  be  shown  to  reliably 
predict  future  values  of  the  controlled  variable  (glucose). 
However,  in  cases  where  the  model  is  less  than  perfect,  it 
has  been  argued5  that  other  control  strategies  may  work 
equally  well  or  better.  The  ability  of  a  model  to  predict 
glucose  excursions  using  meal  and  insulin  information 
likely  depends  on  the  interday  and/or  intraday  variability 
in  the  parameters  of  the  model.  This  is  consistent  with 
the  conclusion  arrived  at  in  the  2008  GMWG  meeting1 
that  physiological  variance  represents  the  single  biggest 
technical  challenge  to  creating  accurate  simulation  models. 
In  this  year's  meeting,  we  attempted  to  expand  on  this  topic 
by  quantifying  the  extent  of  the  variance  and  discussing 
some  of  the  underlying  sources.  Intraday  variance  in 
insulin  sensitivity  was  thought  to  effect  changes  in 
fasting  glucose  of  >15%  in  21  of  23  respondents  (91%), 
with  12  respondents  (52%)  believing  the  variability  to  be 
>30%.  To  counter  these  changes,  respondents  generally 
believed  that  basal  rates  either  need  to  be  adjusted  by 
>30%  (8  of  22  respondents)  or  somewhere  between  15% 
and  30%  (10  of  22  respondents).  Exercise  was  widely 
believed  to  be  one  of  the  underlying  factors  responsible 
for  changing  insulin  requirements,  with  78%  (18/23)  of 
the  respondents  believing  that  exercise  affects  both  the 
needed  insulin  amount  and  acting  time  (four  additional 
respondents  believed  the  interaction  between  exercise 
and  model  parameters  to  be  more  complex  than  simple 
changes  in  insulin  magnitude  and  time  of  effect).  Similarly, 
the  effect  of  a  high-fat  meal  was  perceived  to  be  more 
complex  than  putative  changes  in  gastric  emptying6-9 
by  more  than  two-thirds  of  the  respondents,  with 
approximately  one-half  of  the  respondents  believing  that 
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metabolic  parameters  could  be  affected  for  >12  h  after 
the  meal.  Almost  80%  of  the  participants  thought  that  a 
closed-loop  control  algorithm  should  be  able  to  compensate 
for  these  changes  with  little  (12  of  25  respondents)  or  no 
(7  of  25  respondents)  error. 

Interestingly,  when  asked  about  "insulin  sensitivity  factors" 
commonly  used  in  open-loop  therapy,  there  was  no 
consensus  whether  a  bolus  is  expected  to  effect  a  transient 
or  steady  state  decrease  in  glucose  given  that  the  basal 
rate  is  appropriately  adjusted  to  maintain  glucose  levels 
at  a  stable  level  (specified  as  130  mg/dl  in  question  10). 
Essentially,  the  vast  majority  of  respondents  (82%  [18/22]) 
indicated  that  the  effect  could  not  be  classified  as  transient 
or  fixed  in  all  cases  (of  the  4  respondents  who  chose  one 
of  the  two  outcomes,  the  responses  were  equally  divided). 
What  makes  the  response  interesting  is  that  most  models 
predict  the  response  to  be  transient  (KADIS10,11  being 
a  notable  exception;  see  the  discussion  in  reference  12). 
This  is  one  of  the  few  model  characteristics  that  can  be 
determined  by  examining  the  model  equations  alone. 
If  it  is  true  that,  after  the  basal  rate  is  adjusted  to  achieve 
stable  glucose,  administering  a  bolus  of  insulin  can  lead 
to  a  steady  state  decrease  in  blood  glucose  in  some 
subjects — whereas  in  other  subjects,  the  bolus  has  only 
a  transient  effect — then  models  need  to  be  configured  to 
capture  this  variability. 

One  of  the  more  obvious  differences  among  competing 
metabolic  models  is  their  order,  which  is  defined  as 
the  number  of  differential  equations  used  to  describe 
changes  in  glucose.  This  was  generally  acknowledged  to 
be  a  complex  issue.  Only  30%  (8/26)  of  the  respondents 
thought  that,  if  open-loop  profiles  were  equally  well 
described  by  two  models  with  different  order,  then 
simulated  closed-loop  profiles  would  be  close;  65%  (17/26) 
of  the  respondents  indicated  that  the  question  could 
only  be  answered  by  an  analysis  of  the  different  model 
structures.  Observability  analysis13  applied  to  the  linear 
components  of  the  different  models,  or  to  linearized  versions 
of  nonlinear  components,  was  discussed  as  a  potential 
method  to  assess  the  need  for  high-order  model  terms. 
By  definition,  unobservable/near-unobservable  model  states 
(and  associated  model  equations)  have  no/little  effect  on 
the  model's  output.  What  most  respondents  did  agree  was 
important  is  that  the  pharmacokinetic /pharmacodynamic 
(PK/PD)  response  obtained  from  a  sample  of  virtual 
subjects  should  have  peak  concentration  and  effect 
times  not  statistically  different  from  published  values 
(46%  [12/26]  of  the  respondents)  and  that  cases  where 
the  response  differs  by  >3  standard  deviations  from  the 
mean  should  be  separately  identified  and  analyzed 


(42%  [11/26]  of  the  respondents).  The  PK/PD  profiles 
have  been  obtained  clinically  and  characterized  for 
virtually  all  insulin  types  and  mixes,14-17  by  subject  age,18 
and  by  the  number  of  days  of  catheter  use19  (pump). 
One  might  argue  that  other  key  clinical  observations  should 
also  be  reproducible  by  model  simulation.  For  example, 
one  could  ask  whether  the  model's  response  to  a  short 
interruption  in  insulin  delivery  should  also  be  similar  to 
the  response  observed  clinically,20  as  an  interruption  in 
pump  delivery  is  putatively  believed  to  be  a  first  step 
in  transitioning  from  open-  to  closed-loop  control  and 
is  widely  accepted  to  be  critical  to  the  overall  safety  of 
closed-loop  devices.21  Obtaining  a  given  model's  PK/PD 
profile,  or  its  response  to  an  interruption  in  pump  insulin 
delivery,  is  relatively  trivial  if  the  model  equations  and 
parameters  are  known;  however,  the  question  remains 
as  to  who  would  do  such  simulations  and  how  results 
that  differ  substantially  from  those  observed  in  the 
clinical  setting  would  be  perceived,  given  that  many 
clinical  studies  are  now  approved  based  on  model 
simulations.2 

As  with  the  first  GMWG  meeting,1  participants  were 
careful  to  distinguish  between  models  that  are  to  be 
used  for  simulation  and  models  that  are  to  be  used  for 
control  (see  the  discussion  in  the  first  GMCW  report  for 
definitions1).  A  class  of  data-driven  models  was  also 
discussed  at  this  year's  meeting,  with  the  primary  question 
being  whether  the  glucose  system  is  stationary.  This  is 
an  important  consideration  in  establishing  the  ability  of 
linear  autoregressive  (AR)  models  to  improve  short-term 
glucose  prediction  accuracy  (i.e.,  glucose  predictions  in 
the  30-60  min  window).  On  this  item,  the  majority  of  the 
respondents  were  of  the  opinion  that  glucose  signals 
are  either  not  stationary  (14  of  20  respondents)  or,  at  best, 
weakly  stationary  (6  of  20  respondents).  Generally,  stable 
AR  models  are  only  representative  of  stationary  processes, 
but  it  may  be  possible  to  mitigate  the  problem  by 
detrending  the  signal — for  example,  by  computing  the 
difference  between  two  consecutive  data  points — before 
developing  an  AR  model.  What  is  "at  stake"  is  the  ability 
to  use  these  models  to  improve  short-term  predictions 
compared  with  other  modeling  approaches  (e.g.,  Kalman 
filter22)  or  the  first-order  finite  difference  methods  commonly 
used  in  existing  CGM  products  (i.e.,  predictions  based 
on  the  assumption  that  future  values  can  be  approximated 
by  the  most  recent  value  plus  the  glucose  rate  of  change 
times  the  desired  future  interval).  Model-based  approaches 
have  substantial  potential  to  improve  hypoglycemic 
detection  rates,  although  it  is  interesting  to  note  that 
40%  of  the  respondents  indicated  that  there  is  no  real 
consensus  on  what  constitutes  a  hypoglycemic  event. 
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Conclusions 

As  in  the  first  GMWG  meeting,1  there  was  widespread 
agreement  on  what  the  unanswered  modeling  questions 
are  but  substantially  less  agreement  on  how  to  answer 
the  questions.  The  model-related  questions  and/or  needs 
believed  to  be  unanswered  or  unmet  include  the  following: 

1.  The  need  for  a  mechanism  that  would  allow  investigators 
to  simulate  different  models  (see  question  14  of  the  survey). 
The  key  question  was  how  an  investigator  can  validate 
a  model  before  using  it  to  make  informed  decisions 
regarding  the  choice  of  a  control  algorithm,  optimal 
tuning  of  a  given  control  algorithm,  estimating  the 
range  of  stability  for  control  parameters,  and  assessing 
closed-loop  stability  as  subject  parameters  change. 

2.  The  need  for  models  to  include  realistic  interday  and 
intraday  variance  in  insulin  requirements.  This  variance 
was  generally  believed  to  lead  to  substantial  changes 
in  the  required  basal  rates  (questions  6,  7,  10,  and  11), 
with  the  effect  of  a  high-fat  meal  and/or  prior  exercise 
possibly  contributing  to  the  variability  (questions  12  and 
13).  The  key  question  is  of  how  to  compare  closed-loop 
simulations  performed  on  models  that  do  not  include 
this  variability  with  clinical  studies  in  which  the 
variability  can  be  substantial. 

3.  Insulin  PK/PD  profiles  associated  with  different  models 
were  widely  thought  to  be  an  important  model 
characteristic  that  could  affect  how  different  control 
algorithms  perform  (question  15).  The  key  issue  again 
being  how  to  interpret  closed-loop  simulations  on 
virtual  subjects  with  PK/PD  responses  that  are 
substantially  different  from  published  studies. 

4.  Continuous  glucose  monitoring  accuracy  was  believed 
by  at  least  one-half  of  the  participants  (questions  1  and  5) 
to  still  be  the  primary  obstacle  in  achieving  closed- 
loop  control,  with  the  inherent  properties  of  ISF  still 
needing  to  be  resolved  (questions  2  and  9)  and  the 
statistical  properties  of  the  glucose  signal  (i.e.,  mean, 
variance,  and  autocorrelation)  still  needing  to  be 
characterized  (question  4).  The  key  question  was  how  to 
evaluate  models  for  their  ability  to  improve  predictive 
alerts  that  warn  subjects  of  impending  hyperglycemic  or 
hypoglycemic  conditions  (also  ill  defined;  see  question  3). 

Questions  identified  at  this  year's  meeting  reflect  many  of 
the  same  issues  discussed  at  the  2008  GMWG  meeting. 
The  need  for  investigators  to  have  access  to  different  models 
was  identified  at  the  2008  meeting  as  a  key  recommendation 
but  was  generally  accepted  at  this  year's  meeting  to 
have  been  unrealistic.  The  need  for  models  to  reflect  the 


variability  in  insulin  requirements  commonly  observed 
in  clinical  practice23,24  continues  to  be  an  issue.  The  issue 
as  to  whether  a  simulation  model's  insulin  PK/PD  profile 
needs  to  be  similar  to  measured  profiles14-19  was  a 
new  argument  this  year,  but  it  can  be  seen  as  a  simple 
example  of  why  individual  investigators  seek  access  to 
model  parameters.  The  profile  was  widely  viewed  as 
an  important  consideration  for  designing  a  closed-loop 
algorithm  (as  also  true  for  an  open-loop  algorithm). 
The  PK/PD  profiles  can  now  be  simulated  for  a  limited 
set  of  patients  identified  with  the  Cambridge  simulator,3 
as  the  developer  of  that  model  allows  access  to  individual 
parameters  under  a  research  license  agreement  (personal 
communication  from  R.  Hovorka).  In  addition,  all  subjects 
identified  to  date  using  the  Medtronic  Virtual  Patient 
(MVP)  model  are  now  publicly  available,4  but  this  only 
includes  10  adult  subjects  studied  under  that  company's 
closed-loop  controller.25  Similarly,  a  limited  subset  of 
patient  parameters  are  available  for  the  UVA  simulator.2 

The  positive  outcome  of  this  year's  meeting  was  that  all 
the  participants  expressed  interest  in  attending  a  new 
workshop  devoted  to  presenting  and  discussing  their 
models.  David  Klonoff  (editor-in-chief  of  this  journal  and 
a  key  individual  responsible  for  organizing  the  meeting) 
has  written  an  article  highlighting  how  such  a  workshop 
might  be  conducted.26  Moreover,  given  the  interest  by 
the  GMWG  participants  and  the  subsequent  positive 
feedback  from  senior  leadership  in  distinct  organizations, 
the  U.S.  Army  is  considering  sponsoring  such  an  event 
for  2011.  While  many  of  the  details  obviously  need  to 
be  finalized,  one  of  the  key  ideas  being  considered  is 
to  make  new  data  available  for  investigators  to  identify 
(i.e.,  customize)  their  models.  This  would  eliminate  the 
need  for  any  of  the  investigators  to  divulge  proprietary 
information  and  still  enable  a  limited  evaluation  and 
comparison  of  the  individual  approaches  using  common 
data  sets  and  testing  metrics.  Four  data  sets  are 
recommended  in  the  article  written  by  Klonoff.26  Although 
four  data  sets  are  unlikely  to  have  the  statistical  power 
to  detect  differences  in  closed-loop  simulation  results, 
compared  with  one  another  or  with  results  obtained 
in  published  clinical  studies,25,27-29  the  approach  should 
serve  to  delineate  broad  differences  in  how  the  models 
perform.  It  is  also  unlikely  that  models  used  for  simulation 
can  be  identified  from  such  data,  as  the  models  are 
typically  of  high  order  and  may  require  glucose  tracers 
to  be  identified.1  While  this  would  not  allow  the  UVA 
simulator,  by  definition  a  simulation  model,  to  be  evaluated, 
it  is  likely  that  the  Cambridge3  and  the  KADIS11  models 
could  be  identified  by  the  developers  of  those  models 
using  their  proprietary  software,  and  anyone  familiar 
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with  systems  identification  could  identify  parameters 
for  the  MVP  model.  Noteworthy  is  that  neither  the 
KADIS  nor  the  MVP  model  conveniently  fit  into  the 
"models  used  for  simulation/models  used  for  control" 
categorization,  with  the  KADIS  model  being  something 
of  a  hybrid,  allowing  individual  patients  to  be  identified 
and  simulations  to  be  performed  to  optimize  open-loop 
control,10'11  and  the  MVP  model  used  in  a  similar  manner 
but  with  optimization  done  on  closed-loop  control.30 
Similarly,  the  low-order  models  used  for  control,  for 
example,  the  glucose  model31  with  multiple  competing 
insulin/meal  models  used  to  effect  control  in  the  study 
by  Hovorka  and  colleagues,29  might  be  evaluated  using 
such  data  (as  could  the  internal  models  of  virtually  all 
the  proposed  MPC  algorithms).  Finally,  all  modeling 
approaches  proposed  for  improving  the  prediction  of 
glucose  values  from  CGM  data  per  se  could  be  compared, 
e.g.,  the  Wiener,30  AR,32  and  Kalman22  approaches. 

In  conclusion,  the  survey  responses  obtained  at  the  2009 
GMWG  meeting  indicate  a  lack  of  consensus  by  many  of 
the  participants  on  multiple  issues,  suggesting  that  the 
approaches  to  glycemic  modeling  and  control  are  still 
fluid.  The  GMWG  continues  to  look  for  an  approach  to 
validate  and  compare  models,  with  the  objective  being 
to  highlight  what  the  strengths  of  the  different  models 
are  and  elucidate  gaps  and/or  weaknesses.  Although  we 
cannot  disagree  with  the  desire  of  individual  investigators 
to  keep  the  details  of  their  models  confidential,  the  authors 
of  this  report  continue  to  advocate  open  collaboration 
between  groups  as  well  as  the  need  for  models  to  undergo 
independent  validation  before  being  used  as  the  sole 
mechanism  to  make  control  algorithms  decisions  or 
determine  the  underlying  stability  and  safe  operating 
range  of  a  given  control  algorithm.  With  these  objectives  in 
mind,  a  new  workshop  is  being  planned,  the  U.S.  Army- 
sponsored  Modeling  Comparison  Workshop,  which  will 
focus  on  comparing,  contrasting,  and  understanding 
the  differences  among  the  different  models  and  control 
algorithms.  This  workshop  is  planned  for  2011. 
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Appendix:  Meeting  Survey  with  Responses  Shown  in  Brackets 


1)  Can  currently  available  continuous  glucose  monitoring  (CGM)  devices  provide  accurate  inference  of  blood 

(i.e.,  plasma)  glucose  concentrations  for  the  purpose  of  closed-loop  control? 

a.  [5]  Yes,  it  is  not  a  problem. 

b.  [11]  Accurate  inference  can  only  be  achieved  via  calibration  and  postsignal  processing  techniques,  which  are 
still  being  researched. 

c.  [11]  The  accuracy  of  the  current  available  CGM  devices  must  be  improved  before  the  devices  are  used  for  closed- 
loop  control. 

2)  Subcutaneous  interstitial  fluid  (ISF)  glucose  is  typically: 

a.  [9]  Delayed  by  >15  min,  and  the  plasma-to-ISF  glucose  gradient  is  increased  by  insulin. 

b.  [7]  Well-equilibrated  with  plasma  glucose  within  10-15  min,  but  changes  in  plasma  insulin  may  lead  to  errors 
in  sensor  glucose  of  >10%. 

c.  [7]  Delays  in  ISF  glucose  and  changes  in  the  plasma-to-ISF  glucose  gradient  do  not  lead  to  errors  in  glucose 
sensing  of  >5-10%,  but  errors  in  obtaining  accurate  reference  glucose  values,  changes  in  sensor  sensitivity,  and 
other  factors  can  still  be  a  problem. 

3)  Is  there  a  consensus  of  what  constitutes  hypo-  and  hyperglycemic  episodes? 

a.  [1]  Yes  (e.g., _ consecutive  minutes  of  CGM  signals  below/above  a  threshold _ minutes  apart  from  the 

previous  episode). 

b.  [11]  No. 

c.  [15]  To  some  extent. 

4)  Is  the  maximum  rate  of  increase/decrease  of  subcutaneous  glucose  levels  well  characterized? 

a.  [8]  Yes  (e.g.,  +/- _ mg/dl  Vmin4),  4  being  the  average  answer. 

b.  [8]  No. 

c.  [11]  To  some  extent. 

5)  What  is  the  biggest  impediment  for  the  development  of  an  artificial  pancreas? 

a.  [9]  Lack  of  high-fidelity  predictive  models. 

b.  [4]  Complete  understanding  of  what  the  control  strategies  should  be. 

c.  [13]  Unreliable  measurements  of  glucose  concentration  levels. 
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6)  With  a  fixed  basal  rate,  changes  in  day-to-day  insulin  sensitivity  result  in  morning  fasting  glucose  that 

a.  [2]  Typically  is  within  15%  if  the  subject  has  accurately  administered  insulin  for  all  meals  during  the  previous 
day  and  not  eaten  for  10  h. 

b.  [9]  Typically  may  be  15-30%  different  even  if  the  subject  has  accurately  administered  insulin  for  all  meals 
during  the  previous  day  and  not  eaten  for  10  h. 

c.  [12]  May  be  >30%  even  if  the  subject  has  accurately  administered  insulin  for  all  meals  during  the  previous  day 
and  not  eaten  for  10  h. 

7)  Day-to-day  changes  in  insulin  sensitivity  typically  require 

a.  [4]  Changes  in  overnight  basal  rates  of  up  to  15%  to  achieve  target. 

b.  [10]  Changes  in  overnight  basal  rates  of  up  to  15-30%  to  achieve  target. 

c.  [8]  Changes  in  overnight  basal  rates  of  >30%  to  achieve  target. 

8)  Closed-loop  controllers  should  be 

a.  [15]  Designed  to  adapt  to  changes  in  the  patient  (or  model  parameters)  automatically,  with  no  physician-based 
adjustments. 

b.  [5]  Adjusted  by  physicians  in  much  the  same  way  that  open-loop  algorithms  are  adjusted,  with  more  complicated 
self-tuning  algorithms  introduced  at  a  later  point. 

c.  [8]  The  strategy  that  works  best  based  on  existing  model  simulations  should  be  introduced  first. 

9)  Are  ISF  glucose  signals 

a.  [0]  Stationary. 

b.  [6]  Weakly  stationary. 

c.  [14]  Not  stationary. 

10)  Given  a  well-controlled  individual  with  bedtime  and  morning  glucose  equal  to  130  mg/dl  and  an  insulin  sensitivity 

factor  of  "1  U  drop  of  glucose  30  mg/dl," 

a.  [2]  A  1  U  nighttime  (10  PM)  bolus  of  insulin  is  expected  to  temporarily  lower  blood  glucose  to  100  mg/dl,  but 
morning  glucose  (10  h  later)  is  expected  to  return  to  a  value  near  130  mg/dl.  To  effect  a  change  in  the  morning 
value  requires  a  change  in  overnight  basal. 

b.  [2]  A  1  U  nighttime  bolus  of  insulin  is  expected  to  lower  blood  glucose  by  approximately  30  mg/dl,  with 
morning  glucose  also  lower  by  30  mg/dl.  No  change  in  basal  is  required,  as  the  subject  is  already  stable  at  the 
existing  basal  rate. 

c.  [18]  Neither  statement  "a"  nor  "b"  is  true  in  all  cases. 
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11)  How  well  does  a  closed-loop  controller  need  to  adjust  to  changes  in  insulin  sensitivity? 

a.  [7]  The  controller  should  adapt  to  changes  in  sensitivity  such  that  fasting  glucose  is  at  target  when  the  sensor  is 
correctly  calibrated. 

b.  [12]  Fasting  glucose  should  be  within  15  mg/dl  of  target  when  the  sensor  is  accurate.  Target  should  be  thought 
of  as  a  "range,"  not  a  value. 

c.  [6]  Fasting  glucose  should  be  within  15-30  mg/dl  of  target  when  the  sensor  is  accurate. 

12)  Exercise  can 

a.  [18]  Change  insulin  requirements  for  12-24  h  after  the  exercise.  Both  the  absolute  amount  needed  and  the  rate 
at  which  insulin  acts  can  change. 

b.  [1]  Change  insulin  requirements  for  12-24  h  after  the  exercise;  however,  only  the  absolute  amount  of  insulin 
needed  changes,  with  other  metabolic  parameters  being  largely  unaffected. 

c.  [0]  Neither  statement  "a"  nor  "b"  is  true.  The  effect  that  exercise  is  expected  to  have  on  model  parameters  is 
minor. 

d.  [4]  Neither  statement  "a"  nor  “b"  is  true.  The  effect  that  exercise  is  expected  to  have  on  model  parameters  is 
much  more  complex. 

13)  A  high-fat  meal  can 

a.  [5]  Affect  gastric  emptying  but  has  little  effect  on  other  metabolic  parameters,  such  as  insulin  sensitivity  or 
endogenous  glucose  production. 

b.  [10]  Effect  changes  in  metabolic  parameters  other  than  gastric  emptying,  with  the  effect  lasting  12  h  or  more. 

c.  [0]  Neither  statement  "a"  nor  "b"  is  true.  The  effect  of  a  high-fat  meal  on  model  parameters  is  minor. 

d.  [6]  Neither  statement  "a"  nor  "b"  is  true.  The  effect  of  a  high-fat  meal  on  model  parameters  is  much  more 
complex. 

14)  Given  low-  and  high-order  models  with  parameters  chosen  such  that  the  24  h  open-loop  predictions  are  within 

10%  at  all  time  points,  the  closed-loop  simulation  results  would  be  expected  to  be 

a.  [8]  Nearly  similar. 

b.  [1]  Dramatically  different,  with  one  model  predicting  some  controllers  to  be  unstable  and  unsafe  and  the  other 
models  predicting  the  same  controller  to  be  both  stable  and  safe. 

c.  [17]  Neither  "a"  nor  "b"  can  be  said  to  be  true,  with  the  answer  depending  on  how  the  low-  and  high-order 
models  are  structured 
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15)  Should  a  model  composed  of  a  population  of  "virtual  subjects"  generate  pharmacokinetic/pharmacodynamic  (PK/PD) 

responses  not  statistically  different  from  those  in  the  literature? 

a.  [12]  Yes,  the  average  time-to-peak  concentration  and  peak  effect  should  not  be  statistically  different  for  measured 
values. 

b.  [3]  No,  the  simulation  model  should  represent  extreme  cases  that  can  potentially  be  encountered  in  the  real 
population. 

d.  [11]  Both  should  be  done,  but  patients  whose  PK/PD  response  are  >3  standard  deviations  from  the  mean  should 
be  identified. 
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